Using physical objects in conjunction with an interactive surface

ABSTRACT

An interaction management module (IMM) is described for allowing users to engage an interactive surface in a collaborative environment using various input devices, such as keyboard-type devices and mouse-type devices. The IMM displays digital objects on the interactive surface that are associated with the devices in various ways. The digital objects can include input display interfaces, cursors, soft-key input mechanisms, and so on. Further, the IMM provides a mechanism for establishing a frame of reference for governing the placement of each cursor on the interactive surface. Further, the IMM provides a mechanism for allowing users to make a digital copy of a physical article placed on the interactive surface. The IMM also provides a mechanism which duplicates actions taken on the digital copy with respect to the physical article, and vice versa.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a divisional of, and claims priority to, U.S.patent application Ser. No. 12/241,281, filed on Sep. 30, 2008, which isincorporated herein by reference in its entirety.

BACKGROUND

An interactive surface provides an alternative mechanism for interactingwith computer-implemented applications. A typical interactive surfaceincludes a projection mechanism for displaying information on theinteractive surface. The interactive surface also includes an inputmechanism for detecting selections made by a user in the course ofinteracting with the interactive surface. For example, the interactivesurface may detect when the user touches particular parts of theinteractive surface. Through such interaction, the user may manipulatedigital objects which are displayed by the interactive surface andperform other operations.

There remains ample room for improvement in interactive surfacetechnology. For instance, the input mechanism provided by an interactivesurface is sometimes cumbersome to use (because, for instance, users mayhave difficulty entering information or manipulating digital objectsusing this input mechanism). The input mechanism may also provideunsatisfactory precision and responsiveness.

Further, the interactive surface may potentially confuse users. This maybe particularly a problem when multiple users engage the sameinteractive surface. In this scenario, the interactive surface canpotentially become cluttered with multiple digital objects. Further, fora large interactive surface, it may become difficult for users tomanipulate some digital objects that are beyond their reach.

Further, the interactive surface may fail to provide a collaborativeenvironment which accommodates the manner in which users naturallyinteract in groups. For example, the collaborative environment may failto provide a convenient mechanism for sharing information amongparticipants of a meeting; the interactive surface may likewise fail toprovide a suitable mechanism for collaboratively working on sharedinformation.

There may be additional shortcomings of existing interactive surfacetechnology.

SUMMARY

In one illustrative implementation, an interactive management module(IMM) is described for allowing users to engage an interactive surfaceusing physical devices (“devices”). The IMM detects placement of adevice at a selected location and orientation on the interactivesurface. In response, the IMM displays a digital object on theinteractive surface for use in association with the device. The digitalobject can be placed in proximity to the device; further, the digitalobject can have the same orientation as the device. In one case, thedevice is a keyboard-type device and the digital object provides adisplay interface that presents a visual representation of informationentered by a user using the keyboard-type device.

According to another illustrative aspect, the IMM can interact withmultiple physical input devices that are placed on the interactivesurface at the same time, thus accommodating the use of the interactivesurface in a collaborative environment.

According to another illustrative aspect, the IMM displays a visualattribute which associates the device with the digital object. Forexample, the IMM can present borders (or the like) around both thedigital object and the device; the borders can be assigned the samecolor to convey the affiliation between the digital object and thedevice. Alternatively, or in addition, the visual attribute cancorrespond to a line which connects the digital object to the device.

According to another illustrative aspect, a user can move the device inproximity to another existing display object. In response, the IMM canautomatically associate the device with that other display object.

According to another illustrative aspect, the user can move anotherexisting display object in proximity to the device. In response, the IMMcan automatically associate the existing display object with the device.

According to another illustrative aspect, the IMM can detect theplacement of a second device in proximity to the first-mentioned device.In response, the IMM can associate both devices with a single digitalobject. In one particular example, a first user may enter a firstselection using the first device and a second user may enter a secondselection using the second device. Upon bringing the first and seconddevices together, the IMM can merge the selections identified by the twousers, creating a combined selection.

According to another illustrative aspect, the IMM can display a newdigital object at a location on the interactive surface which: a)satisfies at least one placement constraint to an extent deemedappropriate; and b) reduces interference between the new digital objectand other existing objects on the interactive surface.

According to another illustrative aspect, the device is a mouse-typedevice. The IMM operates by determining an absolute position of themouse-type device and an orientation of the mouse-type device on theinteractive surface. The IMM uses this information to define a frame ofreference. The IMM displays a cursor on the interactive surface relativeto the frame of reference that has been defined. The frame of referencecan be selected relative to a nearest edge of the interactive surface(or other reference object associated with the interactive surface) orthe orientation of the mouse-type device itself, and so on.

According to another illustrative aspect, the IMM allows a user toactivate a touch mode of the mouse-type device, whereupon the cursorsimulates a finger contact point.

According to another illustrative aspect, the IMM allows the user tomodify a digital object using any combination of cursor contact pointsand finger contact points. For instance, the user can manipulateout-of-reach digital objects using one or more cursor contact points(e.g., as controlled by multiple input devices that are simultaneouslyplaced on the interactive surface).

According to another illustrative aspect, the IMM can maintain aplurality of image representations of the interactive surface (and/orindividual digital objects and/or physical objects that have been placedon the interactive surface) at different respective points in time. Auser can retrieve these image representations to investigate the historyof operations taken with respect to the interactive surface.

According to another illustrative implementation, an interactivemanagement module (IMM) is described for allowing users to engage aninteractive surface that contains a physical article (“article”), suchas a tangible medium (e.g., a physical document) having informationvisually presented on its surface. In one illustrative implementation,the IMM operates by detecting the placement of an article on theinteractive surface. The IMM then displays an interface object on theinteractive surface for use in association with the article. The IMMdetects a copy-related activation by a user of the interface object,and, in response, generates a digital copy of the article. Thecopy-related activation can correspond to a motion in which the usermetaphorically drags the digital copy off the article by pulling on theinterface object. The IMM deposits the digital copy at a location on theinteractive surface identified by the user.

According to another illustrative aspect, the IMM can generate thedigital copy by taking an image of the entire interactive surface, andthen cropping the image of the entire surface to obtain an image of thearticle.

According to another illustrative aspect, the IMM can detect a user'sselection of a particular part of the digital copy. In response, the IMMcan highlight a corresponding part of the article. The same procedurecan be performed in reverse, e.g., where the user selects a part of thearticle, prompting the IMM to highlight a corresponding part of thedigital copy.

According to another illustrative aspect, the IMM can detect when theuser makes a mark on the digital copy. In response, the IMM can apply acorresponding digital mark to the physical article. The same procedurecan be performed in reverse, e.g., where the user makes a mark on thearticle, prompting the IMM to make a corresponding mark on the digitalcopy.

According to another illustrative aspect, the IMM can detect that theuser has moved the digital copy to an editing region of the interactivesurface. The IMM can next detect that user has applied a physical markto the digital copy within the editing region. In response, the IMM cangenerate another digital copy of the article. This other digital copyincludes a digital representation of the physical mark, along with itsoriginal content.

This Summary is provided to introduce a selection of concepts in asimplified form; these concepts are further described below in theDetailed Description. This Summary is not intended to identify keyfeatures or essential features of the claimed subject matter, nor is itintended to be used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an illustrative interactive surface environment in which aplurality of users can engage an interactive surface.

FIG. 2 shows illustrative content-projection functionality and inputfunctionality that can be used in the interactive surface environment ofFIG. 1.

FIG. 3 shows an illustrative top-down view of an interactive surface foruse in the illustrative interactive surface environment of FIG. 1.

FIG. 4 is a more detailed top-down view of part of the interactivesurface shown in FIG. 3.

FIG. 5 is a flowchart that shows an illustrative procedure fordetermining a position of a cursor on an interactive surface.

FIG. 6 shows a first technique for determining a frame of reference foruse in positioning the cursor in the procedure of FIG. 5; here, theframe of reference is based on a nearest edge of the interactivesurface.

FIG. 7 shows a second technique for determining a frame of reference foruse in positioning the cursor in the procedure of FIG. 5; here, theframe of reference is based on an orientation of a mouse-type devicewhich is used to control the cursor.

FIG. 8 is a flowchart (and accompanying example) that shows anillustrative procedure for generating a digital copy of an article thatis placed on the interactive surface.

FIG. 9 is a flowchart (and accompanying example) that shows anillustrative procedure for placing a digital object on the interactivesurface in a manner such that the digital object does not interfere withexisting objects on the interactive surface.

FIG. 10 is a flowchart that shows an illustrative procedure foroperating a mouse-type device in a touch mode; in this mode, a cursorcontrolled by the mouse-type device simulates a finger contact placed onthe interactive surface.

FIG. 11 is a flowchart (and accompanying example) that shows anillustrative procedure for generating a new digital object when a deviceis placed on the interactive surface.

FIG. 12 is a flowchart (and accompanying example) that shows anillustrative procedure for affiliating an existing digital object with adevice when the device is placed on the interactive surface in proximityto the existing digital object.

FIG. 13 is a flowchart (and accompanying example) that shows anillustrative procedure for affiliating an existing digital object with adevice when the existing digital object is moved in proximity to thedevice.

FIG. 14 is a flowchart (and accompanying example) that shows anillustrative procedure for affiliating a single digital object with twodevices; this procedure also demonstrates how content provided by tworespective digital objects (controlled by the two respective devices)can be merged by bringing the devices in proximity to each other.

FIG. 15 is a flowchart (and accompanying example) that shows anillustrative procedure for manipulating a digital object using differentcontact point scenarios.

FIG. 16 is a flowchart (and accompanying example) that shows anillustrative procedure for manipulating a digital object by placing adevice on top of the digital object and metaphorically using the deviceas a handle to move the digital object or perform some other action onthe digital object.

FIG. 17 is a flowchart (and accompanying example) that shows anillustrative procedure for creating a digital copy based on a physicalarticle that is placed on the interactive surface.

FIG. 18 is a flowchart (and accompanying example) that shows anillustrative procedure for presenting a menu object in association withan article that is placed on the interactive surface.

FIG. 19 is a flowchart (and accompanying example) that shows anillustrative procedure for highlighting a part of an article based on aselection of a corresponding part of a digital object (or vice versa).

FIG. 20 is a flowchart (and accompanying example) that shows anillustrative procedure for creating a digital mark on a physical articlebased on a corresponding digital mark that is made to a digital copy.

FIG. 21 is a flowchart (and accompanying example) that shows anillustrative procedure for applying a physical mark to a digital copy ofan article within an editing region of the interactive surface, followedby generating a new digital copy of the article which includes a digitalrepresentation of the mark.

FIG. 22 is a flowchart that shows an illustrative procedure forgenerating a series of representations of an interactive surface (and/ora series of representations of any individual object that has appearedon the interactive surface) at different respective times, anddisplaying any information regarding this series to a user.

FIG. 23 shows illustrative processing functionality that can be used toimplement any aspect of the features shown in the foregoing drawings.

The same numbers are used throughout the disclosure and figures toreference like components and features. Series 100 numbers refer tofeatures originally found in FIG. 1, series 200 numbers refer tofeatures originally found in FIG. 2, series 300 numbers refer tofeatures originally found in FIG. 3, and so on.

DETAILED DESCRIPTION

This disclosure sets forth an approach that allows users to engage aninteractive surface using physical objects that are placed on theinteractive surface. In one case, the physical objects may correspond tokeyboard-type devices, mouse-type devices, or other types of inputdevices (or any combination of different types of input devices). Thephysical objects may also correspond physical documents (or the like)having information presented on their respective surfaces. The approachalso provides various mechanisms for managing the association betweenphysical objects that are placed on the interactive surface andrespective digital objects.

According to one scenario, the use of physical objects may facilitate auser's engagement with the interactive surface. For example,keyboard-type devices and mouse-type devices exhibit time-testedreliable behavior with which most users are readily familiar; theapproach can leverage these beneficial features in the context of aninteractive surface environment. Further, the approach may provideuseful techniques for disambiguating the correspondence between physicalobjects placed on the interactive surface and digital objects associatedwith the physical objects. This aspect may help reduce confusion whenmultiple users are engaging the same interactive surface. Further, theapproach provides tools which allow users to access and manipulatedigital objects which are out-of-reach on the interactive surface.Further, the approach provides tools for allowing users to convenientlydisseminate information to other users in a collaborative environment;the approach then allows users to collaboratively interact with theshared information in an efficient and user-friendly manner. Moregenerally, the concepts disclosed herein may address one or more of thechallenges or problems previously noted, but are not limited toaddressing all or any of these challenges or problems.

This disclosure is organized as follows. Section A describesillustrative systems and tools that can be used to engage an interactivesurface using physical objects. Section B describes illustrativeapplications of the systems and tools set forth in Section A. Section Cdescribes illustrative processing functionality that can be used toimplement any aspect of the features described in Sections A and B.

As a preliminary matter, some of the figures describe the concepts inthe context of one or more components, variously referred to asfunctionality, modules, features, elements, etc. The various componentsshown in the figures can be implemented in any manner, for example, bysoftware, hardware, firmware, manual processing operations, and so on,or any combination of these implementations. In one case, theillustrated separation of various components in the figures intodistinct units may reflect the use of corresponding distinct physicalcomponents. Alternatively, or in addition, any single componentillustrated in the figures may be implemented by plural physicalcomponents. Alternatively, or in addition, the depiction of any two ormore separate components in the figures may reflect different functionsperformed by a single physical component. FIG. 23, to be discussed inturn, provides additional details regarding one illustrativeimplementation of the functions shown in the figures.

Other figures describe the concepts in flowchart form. In this form,certain operations are described as constituting distinct blocksperformed in a certain order. Such implementations are illustrative andnon-limiting. Certain blocks described herein can be grouped togetherand performed in a single operation, certain blocks can be broken apartinto plural component blocks, and certain blocks can be performed in anorder that differs from that which is illustrated herein (or can beperformed in parallel). The blocks shown in the flowcharts can beimplemented by software, firmware, hardware, manual processing, anycombination of these implementations, and so on.

As to terminology, the phase “configured to” encompasses any way thatany kind of functionality can be constructed to perform an identifiedoperation. The functionality can be configured to perform an operationusing, for instance, hardware, software, firmware, etc., and/or anycombination thereof.

The term “logic” encompasses any functionality for performing a task.For instance, each operation illustrated in the flowcharts correspondsto logic for performing that operation. In one case, logic maycorrespond to computer-readable instructions. In another case, logic maycorrespond to discrete logic components, or a combination ofcomputer-readable instructions and discrete logic components, etc.

A. Illustrative Systems and Tools

A.1. Overview of an Illustrative Interactive Surface Environment (FIGS.1-4)

FIG. 1 shows an illustrative interactive surface environment 100 inwhich a plurality of users 102 can engage an interactive surface 104. Inthe case most commonly evoked herein, the interactive surface 104corresponds to flat horizontal table-top of any size. However, there isno limitation placed on what may constitute the interactive surface 104.In other cases, the interactive surface 104 can be mounted vertically(e.g., on a wall), or at any other non-horizontal orientation.Alternatively, or in addition, the interactive surface 104 can have acurved surface, or any other kind of non-flat contour.

Broadly, any kind of content-projection functionality 106 can be used toproject information onto the interactive surface 104. In one case, thecontent-projection functionality 106 can correspond to one or more videoprojectors which project information onto the interactive surface 104.Such content-projection functionality 106 can be disposed above theinteractive surface 104, below the interactive surface 104, or at anyother orientation with respect to the interactive surface 104 (or at anycombination of orientations). Alternatively, or in addition, thecontent-projection functionality 106 can be integrated with theinteractive surface itself 104. The placement of the content-projectionfunctionality 106 can be selected, in part, based on the constraintsassociated with each application of the interactive surface 104. Forinstance, in certain cases, it may be appropriate to position thecontent-projection functionality 106 above the digital surface 104because physical objects placed on the interactive surface 104 mightotherwise occlude the digital content (e.g., if projected from below).

Input functionality 108 is used to receive input from the users as theusers engage the interactive surface 104. Again, various types andcombinations of input functionality can be used. In one case, the inputfunctionality 108 can correspond to one or more remote sensors thatsense the user's engagement with the interactive surface 104. Forexample, the input functionality 108 can include an infraredemitter-detector arrangement which can detect when a user touches orotherwise moves close to the interactive surface 104. The inputfunctionality 108 can also provide imaging functionality which generatesa high-resolution image of the interactive surface 104 for use as aninput. Any such remote-sensing-type input functionality 108 can bedisposed above the interactive surface 104, below the interactivesurface 104, or at any other orientation with respect to the interactivesurface 104 (or at any combination of orientations). Alternatively, orin addition, the input functionality 108 can be integrated with theinteractive surface 104 itself Again, the placement of the inputfunctionality 108 can be selected, in part, based on the constraintsassociated with each application of the interactive surface 104.Further, as will be described in more detail below, one or more physicalobjects 110 can be placed on the interactive surface 104. A subset ofthese physical objects 110 can correspond to input-type devices. Theseinput devices may receive input from the users as the users interactwith the interactive surface 104, and thus form part of the suite oftechnology that constitutes the input functionality 108. FIG. 2 willprovide additional details regarding one illustrative implementation ofthe content-projection functionality 106 and the input functionality108.

An interaction management module (IMM) 112 corresponding to anyfunctionality for managing the projection of content onto theinteractive surface 104 and the processing of the user's inputs. Morespecifically, the IMM 112 can correspond to technology for performingthe various procedures which will be described below, as illustrated inFIGS. 5-22.

The IMM 112 interacts with one or more application-providing modules 114(referred to in the plural below for brevity). The application modules114 provide any type of functionality that the users can engage, such assearch functionality, document processing functionality, communicationfunctionality (e.g., Email functionality, instant messagingfunctionality, etc.), and so on. The application modules 114 cancorrespond to local application resources, remote (network-accessible)application resources, or some combination thereof. No limitation isplaced on what may constitute an application module as this term is usedherein. Moreover, while many of the examples developed herein willinvolve the use of multiple users 102 interacting with a commonapplication module in collaborative fashion, multiple users can use theinteractive surface 104 to simultaneously interact with differentapplication modules 114. Further, a single user can engage theinteractive surface 104.

Returning to the interactive surface 104 itself, as mentioned above, auser can place any combination of physical objects 110 onto its surface.The term “physical object” encompasses any type of tangible object thatcan be physically handled by the user. A “digital object,” by contrast,encompasses any type of object that is electronically projected onto thesurface of the interactive surface 104 by the content-projectionfunctionality 106. The general term “object” encompasses any class ofobjects, including physical objects and digital objects.

The physical objects 110 described herein encompass two types ofobjects: devices and articles. These terms are labels of convenience inthis explanation and are to be construed broadly. A device encompassesany mechanism that serves a functional role in interacting with theinteractive surface 104. For example, one type of device corresponds toa keyboard-type device which is traditionally used to enter text andother information by manually depressing its keys. Another type ofdevice corresponds to a mouse-type device which is traditionally used tocontrol the movement of a cursor and perform other navigationalfunctions within a graphical user interface (GUI) setting. These areonly two examples. Other input devices can be used to interact with theinteractive surface 104, such as game console-type devices,joystick-type devices, and so on.

The term physical article encompasses any physical object that can beimaged by the input functionality 108 for any purpose. In the case mostcommonly evoked herein, the term physical article corresponds to atangible medium that visually presents information on its surface. Forinstance, in one concrete case, the physical article can correspond to adocument having text and/or graphics and/or any other information on itssurface. For example, the physical article can correspond to a paperdocument or the like. In another case, the physical article cancorrespond to a screen or the like which projects information thereon.In other cases, the article can correspond to any other physical object,such as a three-dimensional model (e.g., a model house, a model car, adoll, etc.), a part of the human anatomy (e.g., a hand), and so on. Nolimitation is placed on what may constitute an article. Indeed, anarticle may correspond to a device as described above. The term“article” will be used instead of “physical article,” except in thoseinstances where it is appropriate to emphasize the distinction between aphysical article and a digital copy of the physical article.

FIG. 2 shows one representative and non-limiting implementation of thecontent-projection functionality 106 and input functionality 108 thatcan be used in the interactive surface environment 100 of FIG. 1. Inthis illustrative example, the interactive surface 104 corresponds to aflat horizontal surface (e.g., as provided by the top of a table). Thephysical objects 110 on its surface can correspond to one or moredevices 202 (e.g., one or more keyboard-type devices, one or moremouse-type devices, etc.), and/or one or more articles 204 to be imaged(e.g., one or more documents, etc.).

The content-projection functionality 106 of FIG. 1 can correspond to oneor more content-projection modules 206 (referred to in the plural belowfor brevity). For example, each of the content-projection modules 206can project a u×v panel of image content onto the interactive surface104. Taken together, the content-projection modules 206 can provide atiled mosaic of such u×v panels that spans the entire surface of theinteractive surface 104. In one implementation, the content-projectionmodules 206 can be disposed above the interactive surface 104,projecting their content down onto the interactive surface 104.

The input functionality 108 of FIG. 1 can correspond to a suite ofdifferent mechanisms to receive information pertaining to a user'sengagement with the interactive surface 104. One such mechanism is oneor more high-resolution imaging modules 208 (referred to below in theplural for brevity). The high-resolution imaging modules 208 cancorrespond to one or more high-resolution video cameras for forming ahigh-resolution image of the surface of the interactive surface 104.Section B will provide examples of circumstances in which the IMM 112can generate images of the interactive surface 104 using thehigh-resolution imaging modules 208. In one implementation, thehigh-resolution imaging modules 208 can be disposed above theinteractive surface 104.

The input functionality 108 can also include one or moreposition-detection modules 210 (referred to in the plural below forbrevity). In one illustrative example, the position-detection modules210 can correspond to one or more infrared (IR) emitter modules 212 inconjunction with one or more IR detector modules 214. The IR emittermodules 212 project electromagnetic signals in the infrared spectrumonto the interactive surface 104. The IR detector modules 214 receivethe IR signals that are reflected by the interactive surface 104 andphysical objects placed on (or near) its surface.

By virtue of changes in the reflectance of IR signals, theposition-detection modules 210 can detect when new physical objects areplaced on the interactive surface 104 or when existing physical objectson the interactive surface 104 are moved (or removed). In oneimplementation, the position-detection modules 210 can be disposed belowthe interactive surface 104. In this example, the interactive surface104 is constructed with a sufficient degree of opacity to “receive”images that are projected by the content-projection modules 206. Inaddition, the interactive surface 104 has a sufficient degree oftransparency such that IR signals projected from beneath the interactivesurface 104 can be used by the position-detection modules 210 inconjunction with the IMM 112 to detect the presence of objects placed ontop of the interactive surface 104. One infrared detection system thatcan be used to implement the position-detection modules 210 is disclosedin U.S. Pat. No. 7,204,428, naming the inventor of Andrew D. Wilson.

The devices 202 are also components of the suite of input functionality108. In one case, the devices 202 can provide input informationcorresponding to key depression data, mouse-click data, mouse positiondata, and so on. The devices 202 can communicate such input informationto the IMM 112 in any manner, such by using wireless communication.Although not shown in FIG. 2, the suite of input functionality 108 canalso include soft-key input mechanisms that are projected by thecontent-projection modules 206 onto the interactive surface 104. Thesoft-key input mechanisms provide touch-selection data that indicateswhen users touch or otherwise engage the soft-key input mechanisms.

Further, the IMM 112 can optionally employ edge detection functionalityand/or shape detection functionality. This functionality analyzes animage of the interactive surface 104 to identify the profiles (contours)associated with the objects. The IMM 112 can apply this analysis to forma more accurate estimate of the positions and orientations of theobjects placed on the interactive surface 104. The IMM 112 can also usethis analysis to help identify the objects (e.g., to distinguishkeyboard-type devices from mouse-type devices, to distinguish a physicalobject from a finger contact point, and so on).

FIG. 3 shows an illustrative top-down view of the interactive surface104 for use in the interactive surface environment 100 of FIG. 1. Inthis example, the interactive surface 104 corresponds to a rectangulartable-top of dimensions M×N. But, as stated, the interactive surface 104can have any shape, surface contour, orientation, and size.

In the particular case of FIG. 3, the interactive surface 104 is largeenough to host a collaborative environment in which a plurality of users(302, 304, 306, 308, 310) can simultaneously engage the interactivesurface 104. As mentioned above, the users may work on the sameapplication provided by a single application module, or the users canwork on separate applications provided by different respectiveapplication modules 114.

Note that the users engage the interactive surface 104 using a pluralityof physical objects. A first class of physical objects corresponds toinput devices. Such devices include keyboard-type devices and mouse-typedevices (but, as stated, the users can use any kind of input device).For example, user 302 is using two mouse-type input devices (312, 314).User 304 is using a keyboard-type device 316 in combination with amouse-type device 318. Similarly, user 306 is using a keyboard-typedevice 320 in combination with a mouse-type device 322. User 308 doesnot operate any physical input device, but he or she can neverthelessmanipulate digital objects using finger contacts (in a manner to bedescribed below). User 310 uses a physical marking implement 324 (e.g.,a pen, etc.) to create marks. A second class of physical objectsincludes articles. For example, user 310 is interacting with a physicaldocument 326.

The interactive surface 104 also provides a collection of digitalobjects which are projected onto its surface by the content-projectionmodules 206. For example, the user 302 is interacting with a digitalcopy 328 of the physical article 326. The user also can interact withcursors (330, 332) which are controlled by the respective mouse-typedevices (312, 314). The digital copy 328 and the cursors (330, 332) aredigital objects insofar as they are not physical objects, but ratherimages projected onto the interactive surface 104 by thecontent-projection modules 206.

The user 304 is interacting with a display interface 334 which serves todisplay the information input by the user 304 using the keyboard-typedevice 316. The user 304 also interacts with an optional “soft key”input mechanism 336 that can be activated by the user 304 by pressing onits key(s) with a finger or stylus or using a cursor-simulated fingercontact, etc. The user also interacts with a cursor 338 which iscontrolled by the mouse-type device 318. Again, all of these objectsconstitute digital objects that are projected onto the interactivesurface 104.

In a similar manner, the user 306 is interacting with a displayinterface 340 that receives information input by the user using thekeyboard-type device 320. The user 306 also interacts with a cursor 342that is controlled by the mouse-type device 322. Further, the user 306can interact with a joint display interface 344 which is shared with theuser 304. That is, the joint display interface 344 can display the unionof content entered by both user 304 and user 306.

User 308, like user 302, is interacting with a digital copy 346 of thephysical article 326.

User 310 is currently not interacting with any digital objects.

FIG. 4 shows a more detailed depiction of the manner in which the user304 can interact with the interactive surface 104. To repeat, the user304 is using a keyboard-type device 316 and a mouse-type device 318 toengage the interactive surface 104. The keyboard-type device 316 is anexample of one type of device for providing alphanumeric text entry. Theinteractive surface 104 provides three digital objects for use by theuser 304, namely a display interface 334, an auxiliary a soft-key inputmechanism 336, and a cursor 338.

In the illustrated scenario, the user 304 is using the keyboard-typedevice 316 to interact with a word processing program. The message thatthe user 304 is typing using the keyboard-type device 316 is displayedin the interface display 334. The user 304 may optionally use thesoft-key input mechanism 336 to provide supplemental instructions thathave a bearing on the task at hand. For example, the word processingprogram (in conjunction with the IMM 112) can provide a soft-key inputmechanism 336 that allows the user 304 to perform actions on the messagethat he or she is typing. For example, the user 304 can use the soft-keyinput mechanism 336 to print the message, send the message to anotheruser, and so on.

In another scenario (not shown), the IMM 112 can use the soft-key inputmechanism 336 to establish a context for use in interpreting theinformation that the user 304 inputs through the keyboard-type device316 (or some other input device). For example, if the user 304 isperforming a search, the soft-key input mechanism 336 can be used toidentify the data store to which the user's 304 search selection is tobe applied. For instance, the soft-key input mechanism 336 can be usedto specify whether the search selection is to be directed to a localdatabase, an organizational database, or a remote public database, andso on.

As can be appreciated from the illustrative scenario shown in FIG. 3, ina multi-user setting, the interface surface 104 can easily becomerelatively cluttered with physical objects and digital objects. It maythus become confusing as to which user controls which object(s). Toaddress this potential challenge, the IMM 112 provides a suite ofmechanisms to help disambiguate “ownership” of objects.

According to one technique, the IMM 112 displays the digital objectsnear the devices which are associated with the digital objects. Forexample, as shown in FIG. 4, the IMM 112 can present the displayinterface 334 directly above the keyboard-type device 316 (although, inother implementations, the IMM 112 can present the display interface 334at other locations relative to the keyboard-type device 316). Further,the IMM 112 can present the digital objects with an orientation thatgenerally matches the orientation of the devices which are associatedwith the digital objects. For example, the orientation of the displayinterface 334 matches the orientation of the keyboard-type device 316.If the user changes the position and/or orientation of the keyboard-typedevice 316, the position and/or orientation of the display interface 334will change in a corresponding manner. Further note that the orientationof the cursor 338 matches the orientation of the mouse-type device 318.

FIG. 4 also shows that the digital objects can be tagged with a visualattribute which helps the user 304 keep track of the digital objectswhich he or she “owns” or controls. For example, the IMM 112 can displaya border 402 around the display interface 334 and a border 404 aroundthe cursor 338. The IMM 112 can also optionally display a border 406around the keyboard-type device 316 and a border 408 around themouse-type device 318. The borders (402, 404, 406, 408) can be displayedin a single color which differs from respective colors assigned to otherusers, thus helping the user 304 keep track of the group of digitalobjects and devices which belong to him or her. The use of color bordersis illustrative of one implementation. Alternatively, or in addition,the entire surfaces of the objects can be tinted in a telltale color.Alternatively, or in addition, the objects can be tagged withdistinguishing information (such as distinguishing textual information,distinguishing symbolic information, etc.), and so on. Alternatively, orin addition, connecting lines can be drawn between the physical devicesand their associated digital objects. For example, as shown in FIG. 4,the IMM 112 can project a line 410 that connects the mouse-type device318 with its associated cursor 338.

The IMM 112 can establish associations between devices and digitalobjects in different ways. According to one representative mechanism,each physical input device can include an identifying tag on itsunderside (e.g., the side which contacts the interactive surface 104).For example, the tag may correspond to a bar code or other kind ofmachine-readable information. The position-detection modules 210 canread this tag to determine the identity of the physical device. Thisinformation, in turn, allows the IMM 112 to correlate the physicaldevice with the digital objects that are associated with the device.Additional information regarding techniques for reading identifyinginformation from devices placed on an interactive surface 104 can befound in U.S. Pat. No. 7,204,428, naming the inventor of Andrew D.Wilson. Returning to FIG. 3, this figure encompasses many differentusage scenarios that are described in detail in Section B below. Beforedelving into those scenarios, the remainder of this section providesadditional information regarding base tools and mechanisms that canallow the IMM 112 to perform the functions that will be described inSection B.

A.2. Illustrative Cursor-Positioning Mechanisms (FIGS. 5-7)

FIG. 5 is a flowchart that shows an illustrative procedure 500 fordetermining a position of a cursor on an interactive surface 104.

In block 502, the IMM 112 determines the orientation and absoluteposition of a mouse-type device (or other type of input device that canbe used to control a cursor). This information can be obtained from anyof the input mechanisms described above, such as the position-detectionmodules 210.

In block 504, the IMM 112 determines a frame of reference for use inpositioning the cursor. More specifically, a typical mouse-type deviceprovides positional offset information that positions a cursor relativeto a frame of reference. Block 504 determines this frame of reference.FIGS. 6 and 7 will provide two illustrative techniques for determiningthe frame of reference.

In block 508, the IMM 112 places the cursor at a position that is basedthe frame of reference determined in block 506. The placement of thecursor thus reflects the contribution of both a global (absolute)measurement (e.g., provided by the position-detection modules 210) and arelative offset measurement (e.g., provided by the mouse-type deviceitself). The relative offset measurement is generally more precise andresponsive compared to the global measurement.

The placement of the cursor may also be based on a user-definablecontrol display (CD) ratio. The CD ratio is a scaling factor thatdefines how far a cursor will advance on the interactive surface 104 inresponse to a corresponding movement of the mouse-type device. In oneillustrative technique, the IMM 112 can display an auxiliarysoft-key-type input mechanism in proximity to the mouse-type inputdevice (not shown) that allows the user to adjust the CD ratio. Forexample, the auxiliary input mechanism can comprise a slider-mechanismor other tunable control mechanism that allows the user to adjust the CDratio.

FIG. 6 shows a first technique for determining a frame of reference foruse in positioning the cursor in the procedure of FIG. 5. In thistechnique, the IMM 112 determines the absolute position of a mouse-typedevice 602 and then determines the edge 604 of the interactive surface104 that is closest to the position of the mouse-type device 602. TheIMM 112 then defines a frame of reference 606 which is based on axesassociated with the nearest edge 606. Alternatively, or in addition, anyother stationary feature(s) of the interactive surface 104 can be usedas a reference object or reference objects, such as one or morereference lines (not shown) on the interactive surface 104.

FIG. 7 shows a second technique for determining a frame of reference foruse in positioning the cursor in the procedure of FIG. 5. In thistechnique, the IMM 112 determines both the absolute position and theorientation of the mouse-type device 602. The IMM 112 then defines aframe of reference 702 which corresponds to the determined orientationof the mouse-type device 602 itself. In this case, the frame ofreference dynamically changes as the user moves the mouse-type device602 on the interactive surface 104.

The two techniques described in FIGS. 6 and 7 are representative; theIMM 112 can apply yet additional techniques or combinations of differenttechniques to position the cursor.

In the above examples, the goal of the procedure 500 is to determine aframe of reference with respect to which relative placement of thecursor can be defined. The IMM 112 can also apply the informationcollected in the procedure 500 to provide a measurement of the absoluteposition of the mouse-type device 602 itself. For example, the IMM 112can use the position-detection modules 210 or other global (i.e.,surface-wide) position sensing mechanism to determine an absoluteposition of the mouse-type device 602. The IMM 112 can then use therelative position measurement provided by the mouse-type device 602 toadjust the above-determined absolute position as the mouse-type device602 is moved.

It is possible to use just the position-detection module 210 todetermine the absolute position of the mouse-type device 602, but it maybe beneficial to determine the absolute position using the combinationof measurements in the manner described above; this is because, forinstance, the relative position measurement provided by the mouse-typedevice may be more responsive (and may potentially have a higherresolution) than the global measurement made by the position-detectionmodules 210. Similar benefits apply to the scenario shown in FIG. 5,where the goal is to determine the relative offset of the mouse-typedevice 602 with respect to a frame of reference. It is possible to usejust the global measurements to determine the movement of the mouse-typedevice 602, but it may be desirable to use a combination of globalmeasurements and local measurements; this is because the localmeasurements provided by the mouse-type device 602 may be moreresponsive and have a greater resolution compared to the globalmeasurements.

A.3. Illustrative Image Acquisition Mechanisms (FIG. 8)

FIG. 8 is a flowchart (and accompanying example) that shows anillustrative procedure for forming a digital copy of an article or otherkind of object that is placed on the interactive surface 104. FIG. 8will be explained in the context of a specific scenario (depicted to theright of the flowchart operations), but it will be appreciated that thisprocedure 800 has general application to any image acquisition scenario.

In block 802, the IMM 112 detects an instruction to generate a digitalcopy of the article. Section B will describe representative occasions onwhich block 802 may be executed. For example, the IMM 112 can executeblock 802 when a user activates an interface object associated with thearticle (e.g., by executing a dragging motion on the interface object).

In block 804, the IMM 112 receives the image of the entire interactivesurface 104. FIG. 8 depicts this image as illustrative “entire surfaceimage” 806. The entire surface image 806 includes a target image 808 asa component part thereof. The target image 808 corresponds to thearticle that is placed on the interactive surface 104.

In block 810, the IMM 112 crops the entire surface image 806 so that allthe image content is effectively discarded except the target image 808.The target image 808 provides a digital copy of the article that canthen be projected back onto the interactive surface 104 in a manner thatwill be described in Section B. The IMM 112 can use different techniquesto crop the entire surface image 806. In one technique, the IMM 112 canuse edge detection to determine the boundaries of the target image 808;the IMM 112 can then remove all of the image content outside theseboundaries using cropping.

A.4. Illustrative Digital Object Placement Mechanisms (FIG. 9)

FIG. 9 is a flowchart (and accompanying example) that shows anillustrative procedure 900 for placing a digital object on theinteractive surface 104 in a manner such that the digital object doesnot interfere with existing objects on the interactive surface 104. Inthe merely representative example of FIG. 9, the IMM 112 may seek toplace an L×M digital object 902 onto the interactive surface 104 suchthat the digital object 902 does not interfere with a collection ofexisting objects 904. For example, assume that the user has just placeda new keyboard-device 906 on the interactive surface 104. The IMM 112responds by placing the digital object 902 onto the interactive surface104. (In the specific scenario of FIG. 9, the display object 902 acts asa display interface for the keyboard-type device 906.) FIG. 9 will beexplained in the specific context of the above-described scenario(involving digital object 902), but it will be appreciated that thisprocedure 900 has general application to any object placement scenario.

In block 908, the IMM 112 determines the characteristics of the digitalobject 902 that it wishes to place on the interactive surface 104. Thecharacteristics of the digital object 902 may correspond to thedimensions of the digital object 902, or more generally, the shape ofthe perimeter of the digital object 902. In the representative exampleof FIG. 9, the IMM 112 determines that the object 902 has dimensions ofL×M.

In block 910, the IMM 112 identifies at least one placement constraint.Each placement constraint corresponds to a rule which acts as aconstraint to be satisfied in placing the digital object 902. Forexample, one rule that may apply to the scenario set forth above is thatthe digital object 902 is to be placed above the keyboard-device 906(relative to the position of the user) and preferably as close aspossible to the keyboard-type device 906 (or a few inches above thekeyboard-type device 906, etc., depending on environment-specificplacement factors). In one implementation, one or more of theseconstraints are defined as mandatory constraints that are required to besatisfied in the placement of the digital object 902. Alternatively, orin addition, one or more of these constraints are defined aspreference-type constraints that are to be satisfied to an extentpossible in the placement of the digital object 902. In yet otherimplementations, the IMM 112 can permit a user to attach a weight toeach constraint which determines the relative importance of theconstraint in the placement of the digital object 902. The IMM 112 canuse these weights to prioritize the various considerations that go intoplacing the digital object 902, particularly in the case in which all ofthe constraints cannot be fully satisfied.

In block 912, the IMM 112 generates a mask (e.g., mask 914) of theentire surface of the interactive surface 104. The IMM 112 can form themask 914 in any way or combination of ways, e.g., by using the positiondetection modules 210 to determine the placement of physical objects onthe interface surface 104. As shown in FIG. 9, the mask 914 reveals thepresence of the collection of pre-existing objects 904 on theinteractive surface 104. These objects 904 may correspond topre-existing physical objects and/or digital objects (note that the IMM112 has a priori knowledge regarding the placement of digital objectsbecause the IMM 112 is the agent which places these objects).Alternatively, or in addition, block 912 can use the high-resolutionimaging modules 208 to form the mask 914.

In block 916, the IMM 112 determines the placement of the digital object902 based on all the information discussed above, namely: a) thecharacteristics of the digital object 902 to be placed (as determined inblock 908); b) the placement constraint(s) (as determined in block 910);and c) the positions of the existing objects 904 on the interactivesurface 104 (as determined in block 912). The IMM 112 can use anyalgorithm for determining the placement of the object 902. In onerepresentative case, the IMM 112 uses an integral image technique todetermine an appropriate placement of the digital object 902 on theinteractive surface 104. The IMM 112 applies the integral imagetechnique by first forming an integral image table in a known manner.The integral image table can be used to investigate a plurality ofplacement possibilities in an efficient manner.

FIG. 9 shows that the L×M display object 902 is placed on theinteractive surface 104 such that it does not overlap with any of theexisting objects 904. Further, this placement at least partiallysatisfies the above-identified placement constraint(s), e.g., becausethe display object 902 is placed relatively close to user'skeyboard-type device 906. However, the placement of the object 902 maynot be optimal (from the standpoint of the placement constraints) due tothe constraining presence of nearby object 918.

A.5. Illustrative Mechanisms for Using a Cursor as a Contact Point (FIG.10)

A contact point corresponds to a point at which the user may engage theinteractive surface 104 to manipulate a digital object that is presentedon the interactive surface 104 or to perform some other action. In onecase, the user can engage the interactive surface 104 by placing afinger (or other body part, such as an entire hand, etc.) or a physicalpointing instrument (such as, a stylus, etc.) on or near the interactivesurface 104. This type of contact point is broadly referred to herein asa finger contact point. In another case, a user can use a cursor whichis controlled by a mouse-type input device as a simulated finger contactpoint. That is, the cursor can operate as a surrogate for the userapplying a finger at the location defined by the cursor. The type ofcontact point associated with a cursor is referred to as a cursorcontact point. Here, the term cursor encompasses any digital object thatcan be manipulated by the user; in one non-limiting case, a cursorcorresponds to an arrow-shaped digital object.

FIG. 10 is a flowchart that shows an illustrative procedure 1000 foroperating a mouse-type device in a touch mode. In the touch mode, thecursor controlled by the mouse-type device simulates a finger contactplaced on the interactive surface 104 in the manner described above.When not acting in the touch mode, the mouse-type device functions in atraditional manner by moving the location of the cursor withoutperforming any supplemental image-manipulation functions.

In block 1002, the IMM 112 determines whether the user has activated thetouch mode of the mouse-type device. In one merely illustrative case,the user can activate this mode by clicking on a specially-assignedmouse button or other control. So long as the user holds this buttondown, the device operates in the touch mode. When the user releases thebutton, the touch mode is de-activated. Consider the example in whichthe user depresses the button of the mouse-type device, and then, withthe button depressed, moves the cursor from a first location to a secondlocation. At the second location, the user releases the button. Thisoperation is functionally equivalent to a user pressing his or herfinger down on the interactive surface 104 and dragging it from thefirst location to the second location.

In block 1004, the IMM 112 operates the mouse-type device in the touchmode if this mode has been activated.

In block 1006, the user may perform any type of image manipulationoperation within the touch mode of the mouse-type device. For example,the user can move a digital object, rotate a digital object, change thesize of a digital object, and so forth. Section B provides examples ofthe types of actions that can be taken when the mouse-type device isoperating in the touch mode of operation.

In block 1008, the IMM 112 detects that the user has de-activated thetouch mode.

In block 1010, the mouse-type device transitions to a traditional modein which it is used to control the movement of the cursor in atraditional manner of operation.

B. Illustrative Applications of Systems and Tools

FIGS. 11-22 provide several scenarios which describe how the systems andtools described in Section A can be used by users to engage theinteractive surface 104. These scenarios are illustrative; the systemsand tools described above can be used to perform many other operationsthat are not specifically enumerated in this section.

B.1. Mechanisms for Affiliating Digital Objects with Physical Objects(FIGS. 11-14)

FIGS. 11-14 show various scenarios that indicate how the IMM 112 canmanage the affiliation between physical devices and digital objects. Inthe scenarios set forth in this subsection, the digital objectscorrespond to display interfaces. These display interfaces are used topresent information that is entered by the users using physical devices,such as keyboard-type devices. However, the principles described hereare applicable to any type of digital object and any type of physicaldevice.

Starting with FIG. 11, this figure is a flowchart (and accompanyingexample) that shows an illustrative procedure 1100 for generating a newdigital object when a device is placed on the interactive surface 104.Namely, in this case, the user has placed a device 1102 onto theinteractive surface 104; it is the goal of the procedure 1100 to displaya digital object 1104 onto the interactive surface 104. In thisillustrative case, the digital object 1104 provides a display interfacefor use in association with the device 1102 (as mentioned above).

In block 1106, the IMM 112 detects that the user has placed the device1102 onto the interactive surface 104. The IMM 112 can make thisdetermination based on input information provided by theposition-detection modules 210, and/or any other input mechanismdescribed above. The IMM 112 can distinguish the device 1102 from aphysical article (such as a physical document) by noting that the device1102 includes a telltale code on its underside. In addition, oralternatively, the IMM 112 can distinguish the device 1102 from thephysical article (or a finger contact point and the like) based oncontour analysis.

In block 1108, the IMM 112 displays the digital object 1104 on thesurface just above the device 1102. More specifically, the placementprocedure 900 described above (in connection with FIG. 9) can be used todetermine the position (and orientation) at which to place the digitalobject 1104. In one case, the IMM 112 can display the digital object1104 immediately after it detects that the user has placed the device1102 onto the interactive surface 104. In another case, the IMM 112 candisplay the digital object 1104 in response to the detection that theuser has begun to interact with the device 1102; for example, the IMM112 can display the digital object 1104 when the user enters a firstkeystroke using the device 1102. Still other factors can play a role indetermining the timing at which to present the digital object 1104. Inone case, the IMM 112 can remove (e.g., turn off) the digital object1104 when it detects inactivity with respect to the device 1102 for aprescribed period of time.

In block 1110, the IMM 112 detects that the user has removed the device1102 from the interactive surface 104.

In block 1112, in response to the removal of the device 1102, the IMM112 removes the digital object 1104 from the interactive surface 104.

FIG. 12 is a flowchart (and accompanying example) that shows anillustrative procedure 1200 for affiliating an existing digital objectwith a device when the device is placed on the interactive surface 104in proximity to the existing digital object. For example, assume thatthe interactive surface 104 already includes a digital object 1202. Inone particular illustrative scenario, this digital object 1202 is beingused by one or more users to interact with an application module. A newuser may wish to join the others and also interact with the applicationmodule via the digital object 1202.

In block 1204, the IMM 112 detects that the new user has placed a device1206 (or other type of input device) onto the interactive surface 104 inproximity to the existing digital object 1202.

In block 1208, the IMM 112 detects whether one or more device-objectaffiliation considerations are satisfied. These conditions determinewhether the new device 1206 is permitted to form an association with thedigital object 1202. One consideration, for example, indicates whetherthe new user has placed the device 1206 close enough to the digitalobject 1202 to warrant automatic association of the device 1206 with thedigital object 1202. Another consideration indicates whether the user ispermitted to join in and contribute to the task. This consideration, inturn, can depend on any number of environment-specific factors, such asthe nature of the task being performed, the identity of the user, and soon. No limitation is placed on what may constitute a device-objectaffiliation consideration.

In block 1210, the IMM 112 forms an association between the device 1206and the digital object 1202 if the consideration(s) identified in block1208 are satisfied. FIG. 12 shows that the display interface 1202 can bepresented with a visual attribute 1212 (such as a colored border) toindicate that the digital object 1202 is now associated with the device1206. The IMM 112 can use other techniques to signify device-objectaffiliation, that is, instead of the visual attribute 1212, or inaddition to the visual attribute 1212. For example, the IMM 112 canproject a line (not shown) that connects the device 1206 to the digitalobject 1202. (The same type of device-to-object visual correlationmechanism(s) can applied to the scenario shown in FIG. 11, although notshown.)

FIG. 13 is a flowchart (and accompanying example) that shows anillustrative procedure 1300 for affiliating an existing digital objectwith a device when the existing digital object is moved in proximity tothe device. That is, FIG. 13 shows a scenario that is complementary tothe scenario described in FIG. 12. In FIG. 13, the user moves anexisting digital object 1302 to a device 1304, whereas, in FIG. 12, theuser moves the existing device 1206 to the digital object 1202.

In block 1306, the IMM 112 detects that the user (or some other agent)has placed the existing digital object 1302 in proximity to the device1304.

In block 1308, the IMM 112 determines whether the device-objectaffiliation consideration(s) are met. These considerations definewhether it is appropriate to associate the digital object 1302 with thedevice 1304. Exemplary such considerations were described in the contextof the discussion of block 1208 of FIG. 12.

In block 1310, the IMM 112 associates the digital object 1302 with thedevice 1304 if the conditions defined in block 1308 are satisfied. Inone case, the IMM 112 can display a visual attribute 1312 (such as acolored border) to indicate that the digital object 1302 is nowassociated with the device 1304.

FIG. 14 is a flowchart (and accompanying example) that shows anillustrative procedure 1400 for affiliating a single digital object withtwo devices; this procedure also demonstrates how operations beingperformed using two respective digital objects (which are controlled bythe two respective devices) can be merged by bringing the devices nearto each other.

For example, in one scenario, assume that a first user is using a firstdevice 1402 and an associated display object 1404 to interact with asearch application module. A second user is using a second device 1406and associated digital object 1408 to interact with the same searchapplication module. At an initial point of time, these users areinteracting with the interactive surface 104 at locations that arerelatively far apart from each other; further, these two users areperforming independent search operations. Next assume that the usersmove their devices (1402, 1406) relatively close to each other so thatthey can work together on a common search task. For example, these twousers may correspond to users 304 and 306 shown in FIG. 3. In thisscenario, the IMM 112 can be used to combine the operations performed bythese users in the manner described below.

In block 1410, the IMM 112 detects placement of the two devices (1402,1406) in proximity to each other on the interactive surface 104.

In block 1412, the IMM 112 determines whether one or more device-deviceaffiliation considerations are satisfied. These considerations defineone or more rules that govern whether the operations being performed byone user are to be merged with the operations being performed by anotheruser. Exemplary such considerations were described in the context of thediscussion of FIGS. 12 and 13.

In block 1414, the IMM 112 merges the functions being performed by thetwo users if the considerations set forth in block 1412 are satisfied.This merging operation can take different forms depending on variousenvironment-specific factors and other considerations. In the merelyillustrative example shown in FIG. 14, the IMM 112 combines the digitalobjects (1404, 1408) that were separately being used by the devices(1402, 1406) into a single digital object 1416. Both users are permittedto engage the joint display object 1416 using their respective devices(1402, 1406).

Further, the IMM 112 can optionally combine the search selections thatwere independently made by the two users into a joint search selection.For example, assume that the first user has entered the search selection“Seattle” and the second user has entered the search selection “Condo”before these users moved their devices (1402, 1406) close together. Uponmoving close together, the IMM 112 can form a combined search selectionthat represents the logical union of the independent search selections(such as by forming the combined search selection of “Seattle ANDCondo”). (In other scenarios, other rules can determine the manner inwhich separate selections are logically combined together.) The IMM 112can forward this combined search selection to an appropriate applicationmodule and then display the results provided by the application modulein the unified display interface 1416, as graphically illustrated inFIG. 14 by the arrow pointing to the label “(Output of Joint Search).”

To repeat, the specific example developed in FIG. 14 is representativeof many possible scenarios in which digital objects and correspondingoperations can be merged based on various triggering events. In anotherexample, two users can take independent notes at a meeting usingindependent digital objects to display those notes. Upon moving closetogether, the IMM 112 can merge the notes taken by these users into acommon document that is displayed by a common digital object. In thisscenario, the IMM 112 can optionally provide user tags of any type toindicate the contributions made by respective users to the commondocument. In general, the nature of the merging operation that isperformed will depend, in part, on the nature of the functions beingperformed by the users, in combination with one or moreenvironment-specific factors. In one case, the IMM 112 can maintainprivilege information which governs, on a user-by-user basis, theobject-combining and operation-combing behavior of the IMM 112. Forinstance, a meeting administrator may wish to prohibit certain usersfrom contributing to a joint task, thereby locking out the behavior setforth in procedure 1400 for certain users.

FIG. 14 was described in the illustrative context of two users. But theprinciples set forth above can be applied to any number of users whowish to interact with a shared digital object.

B.2. Mechanisms for Manipulating Digital Objects (FIGS. 15 and 16)

The next series of figures shows illustrative techniques formanipulating digital objects. Starting with FIG. 15, this figure is aflowchart (and accompanying example) that shows an illustrativeprocedure 1500 for manipulating a digital object using different contactpoint scenarios. As set forth above, a contact point corresponds to apoint at which a user engages (e.g., makes contact with) the surface ofthe interactive surface 104. This function can be performed in differentways. First, the user can create a contact point by pressing his or herfinger (or other body part) or a physical instrument against theinteractive surface 104; this action defines a finger contact point. Orthe user can create a contact point by activating the touch mode of amouse-type device, as described in procedure 1000 of FIG. 10; thisaction defines a cursor contact point.

In block 1502, the IMM 112 detects that the user has engaged a digitalobject using multiple contact points.

In block 1504, the IMM 112 manipulates the digital object in a mannerinstructed by the user (such as by moving the digital object, changingthe orientation of the digital object, changing the size of the digitalobject, and so on).

FIG. 15 presents three representative (and non-exhaustive) scenariosthat explain how the procedure 1500 can be performed. In scenario A, theuser engages a digital object 1506 using two finger contact points(1508, 1510). In this merely illustrative example, the user can use thefinger contact points to touch two opposing corners of the digitalobject 1506. The user can then manipulate the digital object 1506 bymoving the finger contact points (1508, 1510) in a clockwise orcounterclockwise direction to rotate the digital object 1506 (optionallywith one contact point fixed as a pivot), or by moving the fingercontact points (1508, 1510) towards or away from each other to changethe size of the digital object 1506, and so on.

In scenario B, the user engages the digital object 1506 with a fingercontact point 1512 and a cursor contact point 1514. The user can executeany manipulation described above in the context of scenario A usingthese contact points.

In scenario C, the user engages the digital object 1506 with two cursorcontact points (1516, 1518). The user can execute any manipulationdescribed above in the context of scenario A using these contact points.The type of manipulation shown in scenario C is particularly appropriatein those situations in which the user seeks to manipulate an object thathe or she cannot reach by hand. For example, the user 302 of FIG. 3 canuse two mouse-type devices (312, 314) to control two associated cursors(330, 332). By operating the mouse-type devices (312, 314) in the touchmode, the user can manipulate the digital copy 328 in the mannerdescribed above, even though the digital copy 328 is physically out ofthe reach of the user 302.

Although not shown, a user can also use a single contact point tomanipulate a digital object. For example, the user can use a singlefinger contact point or a single cursor contact point to move anydigital object from a first location on the interactive surface 104 to asecond location.

FIG. 16 is a flowchart (and accompanying example) that shows anillustrative procedure 1600 for manipulating a digital object by placinga device on top of the digital object and metaphorically using thedevice as a handle to move the digital object or perform some otheraction on the digital object.

In block 1602, the IMM 112 detects the movement of the device over adigital object.

In block 1604, the IMM 112 manipulates the digital object using thedevice as a handle, in a manner instructed by the user.

For example, consider the scenario depicted to the right of theflowchart blocks in procedure 1600. Here, a user has placed a mouse-typedevice 1606 over the top of a digital object 1608 of any type that isdisplayed on the interactive surface 104. The user can then manipulatethe digital object 1608 in any manner using the mouse-type device 1606,e.g., by rotating the digital object 1608, sliding the digital object1608 to another position, and so on. The association between themouse-type device 1606 and the digital object 1608 can be broken whenthe user lifts the mouse-type device 1606 off the interactive surface104. Alternatively, or in addition, the handle-like capabilities of themouse-type device 1606 can be activated and deactivated by pressing andreleasing a mouse button that is assigned to this function.

B.3. Mechanisms for Creating and Manipulating Digital Copies (FIGS.17-21)

The next series of flowcharts show various techniques for generatingcopies of any type of physical article, such as, but not limited to, atangible (e.g., “hard-copy”) document. The flowcharts also showtechniques for interacting with the digital copies that have beengenerated. The generation and manipulation of digital copies isparticularly useful in a collaborative environment. For example, asshown in FIG. 3, the user 310 is interacting with a physical article326. The user 310 can generate digital copies of this physical article326 (e.g., digital copies 328 and 346) for distribution to other users(e.g., users 302, 308). This makes it easy for the other users (302,308) to examine the information imparted by the physical article 326,e.g., without requiring the user's (302, 308) to crowd around the user310 to examine the original physical article 326, and without requiringthe users (302, 308, 310) to circulate the physical article 326 amongstthemselves in the traditional temporally-successive manner. Further,actions taken by any user with respect to a digital copy canautomatically cause corresponding operations to be performed on thephysical article, and vice versa. This also facilitates thecollaborative nature of the environment, e.g., by providing a convenientmechanism whereby any user can point out a feature of interest to one ormore other users stationed at different locations about the perimeter ofthe interactive surface 104.

Starting with FIG. 17, this figure is a flowchart (and accompanyingexample) that shows an illustrative procedure 1700 for creating adigital copy 1702 based on a physical article 1704 that is placed on theinteractive surface 104.

In block 1706, the IMM 112 detects that the user has placed the physicalarticle 1704 on the interactive surface 104. This detection operationcan be performed based on the output of the position-detection modules210, or by using some other input mechanism or combination of inputmechanisms. The IMM 112 can distinguish the physical article 1704 from afinger contact point (and other objects) using contour analysis. The IMM112 can distinguish the physical article 1704 from an input device bynoting the lack of a telltale marking on the physical article 1704 thatidentifies an object as an input device.

In block 1708, the IMM 112 presents an interface object 1710 inassociation with the physical article 1704. The interface object 1710 isa digital object which allows the user to interact with the physicalarticle 1704, e.g., to make digital copies of the physical article 1704and perform other operations. In the merely representative example ofFIG. 17, the interface object 1710 corresponds to a halo or a borderthat is formed around the physical object 1704. Any other type ofinterface object can be used to accomplish the functions of procedure1700. For example, another interface object (not shown) can correspondto one or more digital pull tabs that protrude from a corner (orcorners) of the physical article 1704.

In block 1712, the IMM 112 detects that the user has engaged theinterface object 1710 in a manner indicative of an intent to create adigital copy of the physical article 1704. In one case, assume that theuse touches the interface object 1710 with a finger and then, whilemaintaining the finger pressed against the interactive surface 104,drags the finger away from the physical article 1704. The IMM 112 caninterpret this motion as an instruction to create a digital copy of thephysical article 1704. This motion resembles the act of sliding atopmost card off of a deck of cards. The same movement can be performedusing a cursor contact point instead of a finger contact point. Further,to repeat, other interface objects and associated telltale movements canbe used to instruct the IMM 112 to create a digital copy. The IMM 112can detect the telltale movement using any one or more of the detectionmechanisms described above (e.g., using the position-detection modules210 and/or the high-resolution imaging modules 208, etc.).

In block 1714, the IMM 112, having detected the type of telltalemovement described above, creates the digital copy 1702 of the physicalarticle 1704. In one case, the IMM 112 can “deposit” the digital copy1702 at a location at which the user terminates his or herfinger-dragging motion and lifts his or her finger from the interactivesurface 104. Although not shown, the IMM 112 can also allow the userspecify various attributes which govern the copying operation, e.g., byspecifying a number of copies to be produced, a size-scaling ratio to beused, a copying density to be used, and so on.

Different environment-specific considerations can be used to determinethe exact timing at which the IMM 112 captures the image of the physicalarticle 1704. In one non-limiting case, the IMM 112 captures the imageof the physical article 1704 when the user removes his or her fingerfrom the interactive surface 104, e.g., at the end of the draggingoperation. This timing can reduce the likelihood that the IMM 112 willcapture part of the user's body when taking a picture of the physicalarticle 1702.

FIG. 18 is a flowchart (and accompanying example) that shows anillustrative procedure 1800 for presenting a menu object 1802 inassociation with a physical article (e.g., physical article 1804) thatis placed on the interactive surface 104. More specifically, in additionto serving as a tool to perform the copying operation (described in FIG.17), the interface object 1710 can be activated to perform otheroperations, such as the presentation of the menu object 1802.

In block 1804, the IMM 112 identifies an action by the user that isindicative of the user's desire to engage the interface object 1710 toperform a task other than the creation of a digital copy of the physicalarticle 1704. For example, in one merely illustrative case, the user maytap on the interface object 1710 twice in rapid succession with a fingeror with a simulated finger contact point (controlled by a mouse-typedevice). In the example of FIG. 18, the IMM 112 interprets this actionas a request to activate the menu object 1802.

In block 1806, the IMM 112 presents the menu object 1802 to the user.The IMM 112 can display this menu object 1802 in any location, such asat the periphery of the interface object 1710. The menu object 1802 canprovide any number of options for performing any number of operationswith respect to the physical article 1704, such as printing a digitalcopy of the physical article 1704, saving a digital copy of the physicalarticle 1704, and so on. Alternatively, or in addition, the menu object1802 can serve as a tool which allows the user to adjust the settingswhich govern the copying operation (of FIG. 17).

FIG. 19 is a flowchart (and accompanying example) that shows anillustrative procedure 1900 for highlighting a part of a physicalarticle 1902 based on a selection of a corresponding part of a digitalcopy 1904 (or vice versa).

In block 1906, the IMM 112 detects that the user has selected a part1908 of the digital copy 1904. For example, the user may point to theparticular part 1908 of the digital copy 1904 using a finger, a stylus,a cursor, or other contact point.

In block 1910, the IMM 112 can provide highlighting 1912 to a part 1914of the physical article 1902. The part 1914 corresponds to the selectedpart 1908 of the digital copy 1904. The IMM 112 can perform thisfunction by mapping positions in the digital copy 1904 to correspondingpositions in the physical article 1902. The IMM 112 can highlight thepart 1914 of the physical copy 1902 in any way, such as by projecting animage of a border around the part 1914, projecting an arrow (or thelike) which points out the part 1914, projecting a light beam whichbrightens the part 1914, and so on.

In block 1916, the IMM 112 detects that the user has de-selected thepart 1908 of the digital copy 1904, e.g., by removing his or her fingerfrom the part 1908.

In block 1918, the IMM 112 removes the highlighting 1912 from thecorresponding part 1914 of the physical article 1902.

Although not shown, the user can select a part of the physical article1902 in the manner described above, which prompts the IMM 112 to addhighlighting to a corresponding part of the digital copy 1904. Asbefore, the IMM 112 can remove the highlighting when the user de-selectsthe previously selected part of the physical article 1902. In thisscenario, the input functionality 108 can include anotherposition-detection system (not shown) which has the capability ofdetecting actions taken by the user directly above the physical article1902. For instance, the input functionality 108 can include a remotesensing mechanism of any type that is positioned above the interactivesurface 104. Alternatively, or in addition, the input functionality 108can include a touch-sensitive input mechanism that is integrated (e.g.,internal to) the interactive surface 104 itself.

In other cases, the IMM 112 can give the user the option of preservingthe highlighting applied to the physical article 1902 or the digitalcopy 1904 after the user has de-selected a previously selected part. Forexample, instead of adding an impermanent highlighting to the digitalcopy 1904 in the manner described above, the user can add a digital markto the digital copy 1904. This causes the IMM 112 to add a correspondingdigital mark to the physical article 1902. The digital mark applied tothe physical article 1902 could remain after the user has finishedengaging the digital copy 1904 (but it is also possible to give the userthe option of “erasing” digital marks applied to the physical article1904).

FIG. 20 expands on the above-identified concept; namely, FIG. 20 is aflowchart (and accompanying example) that shows an illustrativeprocedure 2000 for creating a digital mark 2002 on a physical article2004 based on a corresponding digital mark 2006 that is made to adigital copy 2008.

In block 2010, the IMM 112 detects that the user has placed the digitalmark 2006 on the digital copy 2008. The user can create this digitalmark 2006 using any mechanism 2012, such as a finger, stylus, cursor,and so on. In the merely illustrative scenario of FIG. 20, the user usesa finger to circle a particular part 1214 of the digital copy 2008. TheIMM 112 can detect the creation of the digital mark 2006 using any oneor more of the detection mechanisms described above (e.g., using theposition-detection modules 210 and/or high-resolution imaging modules208, etc.).

In block 2016, the IMM 112 applies a corresponding digital mark 2002 toa corresponding part 2018 of the physical article 2004. Unlike the caseof FIG. 19, the digital mark 2002 remains on the physical article 2004,although it is also possible to provide a mechanism that allows a userto “erase” such digital marks 2002.

The procedure 2000 shown in FIG. 20 can be performed in reverse, e.g.,by detecting that a user has made a physical mark (or digital mark) tothe physical article 2004, and then, in response, adding a correspondingdigital mark to the digital copy 2008. In this scenario, the IMM 112 candetermine that a physical mark has been added by forming a differenceimage which reflects the difference between the physical article at timet₁ (before the physical mark was added) and time t₂ (after the physicalmark was added). The digital counterpart of the physical mark can beobtained based on this difference image.

FIG. 21 is a flowchart (and accompanying example) that shows anillustrative procedure 2100 for applying a mark 2102 to a digital copy2104 of a physical article (not shown), followed by generating a newdigital copy which includes a digital mark 2108 that is a digitalrepresentation of the physical mark 2102.

In block 2110, the IMM 112 detects that the user has moved a digitalcopy 2104 of a physical article over an editing region 2112. The editingregion 2112, in one illustrative case, may correspond to a white boardsection of the interactive surface 104. Or the editing region 2112 cancorrespond to a piece of paper (or the like) placed on the surface ofthe interactive surface 104. The users can draw on this editing region2112, e.g., using a pen, pencil, marker, etc. The editing region 2112has a defined size. In one case, the IMM 112 can “snap” the digital copy2104 to the size of the editing region 2112 when the digital copy 2104is moved over the editing region 2112, that is, by modifying the size ofthe digital copy 2104 such that it conforms to the size of the editingregion 2112. Once the digital copy 2104 has become associated with theediting region 2112, the digital copy 2104 can remain “attached” to theediting region 2112. Thus, when (and if) the editing region 2112 ismoved on the interactive surface 104 (e.g., by moving a whiteboard orpaper associated with the editing region 2112), the digital copy 2104 ismoved along with it. This mechanism provides another way of moving adigital copy on the interactive surface 104, e.g., by metaphoricallyusing the editing region 2112 as a plate on which to move the digitalcopy 2104.

In block 2114, the IMM 112 detects that the user has created a physicalmark 2102 “on” the digital copy 2104 (e.g., using the detectiontechnique described above with respect to FIG. 20). The physical mark2102 is physically formed on the underlying substrate provided by theediting region 2112. In this merely illustrative case, the user uses aninstrument 2116 to circle a part 2118 of the digital copy 2104.

In block 2120, the IMM 112 generates a new digital copy 2106 of theprevious digital copy 2104. The new digital copy 2106 includes a digitalmark 2108 which is the digital counterpart to the physical mark 2102drawn “on” the previous version of the digital copy 2104. In the exampleof FIG. 21, the user has also moved the new digital copy 2106 away fromthe editing region 2112, which reveals the “naked” original physicalmark 2102 placed on the editing region 2112.

The IMM 112 can perform the operation of block 2118 in various ways.According to one technique, the IMM 112 can take another picture of thedigital copy 2104 (with mark 2102 added thereto), potentially with adifferent exposure to account for the fact that the IMM 112 is nowtaking a picture primarily of an image (with the exception of thephysical mark 2102), rather than the original physical document. Inanother technique, the IMM 112 can momentarily turn off the projectionof the digital copy 2104, which will reveal the naked physical mark2102. The IMM 112 can capture an image of the physical mark 2102 andthen digitally composite the image of the physical mark 2102 with theoriginal digital copy 2104 to generate the new digital copy 2106. Stillother techniques can be used to generate the new digital copy 2106.

The procedure 2100 described above can be repeated one or more times.For each iteration, the user may provide additional annotations to thedigital object in the manner described above.

More generally, any of the techniques described in subsection B.3 can becombined together in any manner.

Moreover, in the above examples, the physical article corresponds to aphysical document, such as a paper document. But the proceduresdescribed above can be applied to any physical object. For example, inanother case, the physical object corresponds to a three-dimensionalobject. In another case, the physical object includes a screen whichdisplays information thereon. In another case, the physical objectcomprises (or includes) a part of the user's anatomy. For example, inthe procedure 1700 of FIG. 17, the user may wish to create a digitalcopy of a document in which the user is physically pointing to aparticular passage in the document. Still other applications arepossible.

B.4. Mechanisms for Capturing a History of Actions and Performing OtherSurface Management Functions

FIG. 22 is a flowchart that shows an illustrative procedure 2200 forgenerating a series of representations of the interactive surface 104 atdifferent respective times, and displaying any information regardingthis series to a user.

In block 2202, the IMM 112 can take a picture of the entire interactivesurface 104 at a time t_(n). The IMM 112 can form this picture indifferent ways, such as by taking a single picture of the interactivesurface 104 with both physical objects and digital objects placedthereon. Or the IMM 112 can form a composite image in the mannerdescribed above in the context of the explanation of FIG. 21, e.g., byturning the projection off, taking a picture of the physical articles onthe interactive surface 104, and then combining that picture with thedigital objects that are projected onto the interactive surface 104.

Different events can trigger the image capture operation. In one case,the IMM 112 periodically captures an image of the interactive surface104 (e.g., at every n seconds or at every n minutes, and so on).Alternatively, or in addition, the user can expressly instruct the IMM112 to capture images at various times of interest. Alternatively, or inaddition, the IMM 112 can automatically capture images when itindependently detects that something noteworthy is happening on theinteractive surface 104, such as when a user adds a new device to theinteractive surface 104, or when a user moves an existing device ordigital object on the interface surface 104, and so on. Upon repetition,block 2202 establishes a plurality of images of the interactive surfacetaken at different respective times. The IMM 112 can archive the imagesin a suitable data store or stores (not shown).

In block 2204, the user can instruct the IMM 112 to show any of theimages that it has previously recorded in block 2202. In one case, theuser can ask the IMM 112 to show a selected image or selected images. Inanother case, the user can ask the IMM 112 to display a sequence of therecorded images in the order in which these images were captured over anidentified time span. In this latter scenario, the IMM 112 can presentthe images in an animated manner or a pseudo (roughly) annotated manner.

Procedure 2200, as described above can be used to form a series ofrepresentations of the entire interactive surface 104. Alternatively, orin addition, the procedure 2200 can be used to form a series ofrepresentations of parts of the interactive surface 104, such asindividual objects placed on the interactive surface 104 (includingphysical objects and/or digital objects), even though the user may nothave expressly requested that these representations be formed). Forexample, the procedure 2200 can be used to show the evolution of changesmade to an individual copy of a physical document or the like. The IMM112 can obtain representations of physical objects using, for example,the masking procedure of FIG. 8 or the like. The IMM 112 has immediateaccess to digital objects that it projects onto the interactive surface104.

The IMM 112 can provide various interface mechanisms for use inconjunction with the procedure 2200 of FIG. 22. For example, the IMM 122can display a container-type digital object (not shown) on theinteractive surface 104. The container-type digital object acts as aninterface to a collection of digital objects (associated with individualitems that have appeared on the interactive surface 104 and/orrepresentations of the entire interface surface 104). The user canorganize the digital objects in the container-type digital object usingone or more views. A first view allows the user to arrange the digitalobjects in a grid format or other spatial organization format. A secondview allows the user to arrange the digital objects in a flip-bookformat, e.g., as a temporal sequence of objects. In one case, the usercan also manipulate any digital object via the container-type digitalobject, such as by scaling the digital object, rotating the digitalobject, and so on.

The IMM 112 can also include mechanisms that allow a user to “clean up”the interactive surface 104, e.g., by removing digital objects from theinteractive surface. In one case, the IMM 112 can provide a commandwhich allows the user to delete (e.g., remove) all digital objects fromthe interface surface 104. In addition, or alternatively, the IMM 112can provide a command that allows a user to delete identified digitalobjects.

C. Representative Processing Functionality

FIG. 23 sets forth illustrative electrical data processing functionality2300 (simply “processing functionality” below) that can be used toimplement any aspect of the functions described above. With reference toFIG. 1, for instance, the type of processing functionality 2300 shown inFIG. 23 can be used to implement any aspect of the interactive surfaceenvironment 100. In one case, the processing functionality 2300 maycorrespond to any type of computing device.

The processing functionality 2300 can include volatile and non-volatilememory, such as RAM 2302 and ROM 2304, as well as one or more processingdevices 2306. The processing functionality 2300 also optionally includesvarious media devices 2308, such as a hard disk module, an optical diskmodule, and so forth. The processing functionality 2300 can performvarious operations identified above when the processing device(s) 2306execute instructions that are maintained by memory (e.g., RAM 2302, ROM2304, or elsewhere). More generally, instructions and other informationcan be stored on any computer-readable medium 2310, including, but notlimited to, static memory storage devices, magnetic storage devices,optical storage devices, and so on. The term “computer-readable medium”also encompasses plural storage devices. The term “computer-readablemedium” also encompasses signals transmitted from a first location to asecond location, e.g., via wire, cable, wireless transmission, etc.

The processing functionality 2300 also includes an input/output module2312 for receiving various inputs from a user (via input modules 2314),and for providing various outputs to the user (via output modules 2316).FIGS. 1 and 2 describe various types of such input modules 2312 andoutput modules 2314 (e.g., corresponding to the input functionality 108and content-projection functionality 106, respectively). The processingfunctionality 2300 can also include one or more network interfaces 2318for exchanging data with other devices via one or more communicationconduits 2320. One or more communication buses 2322 communicativelycouple the above-described components together.

In closing, the description may have described various concepts in thecontext of illustrative challenges or problems. This manner ofexplication docs not constitute an admission that others haveappreciated and/or articulated the challenges or problems in the mannerspecified herein.

More generally, although the subject matter has been described inlanguage specific to structural features and/or methodological acts, itis to be understood that the subject matter defined in the appendedclaims is not necessarily limited to the specific features or actsdescribed above. Rather, the specific features and acts described aboveare disclosed as example forms of implementing the claims.

What is claimed is:
 1. A system comprising: one or more processingdevices; and one or more computer-readable memory devices or storagedevices storing computer-readable instructions which, when executed bythe one or more processing devices, cause the one or more processingdevices to: detect placement of a physical input device on aninteractive surface; determine an absolute position of the physicalinput device and an orientation of the physical input device; define aframe of reference based on the absolute position and the orientation ofthe physical input device; obtain a relative offset measurement from thephysical input device; and display a cursor on the interactive surfacerelative to the frame of reference that has been defined, wherein thecursor is placed based on both the absolute position and the relativeoffset measurement obtained from the physical input device.
 2. Thesystem of claim 1, wherein the computer-readable instructions cause theone or more processing devices to: identify a reference object on theinteractive surface and define the frame of reference based on thatreference object.
 3. The system of claim 1, wherein thecomputer-readable instructions cause the one or more processing devicesto: define the frame of reference relative to the orientation of thephysical input device.
 4. The system of claim 1, wherein thecomputer-readable instructions cause the one or more processing devicesto: display a visual attribute which associates the physical inputdevice and the cursor.
 5. The system of claim 1, wherein thecomputer-readable instructions cause the one or more processing devicesto: detect that a control provided by the physical input device has beenactivated; and operate the physical input device in a touch mode,wherein, in the touch mode, the cursor serves as a simulated contactpoint that engages the interactive surface.
 6. The system of claim 1,wherein the computer-readable instructions cause the one or moreprocessing devices to: identify a stationary feature of the interactivesurface and define the frame of reference based on the stationaryfeature of the interactive surface.
 7. The system of claim 1, furthercomprising: a global position sensing mechanism configured to determinethe absolute position of the physical input device.
 8. A systemcomprising: one or more processing devices; and one or morecomputer-readable memory devices or storage devices storingcomputer-readable instructions which, when executed by the one or moreprocessing devices, cause the one or more processing devices to:determine whether a touch mode of a mouse-type device has beenactivated, the mouse-type device having: the touch mode where themouse-type device is used as a simulated finger contact, and anothermode where the mouse-type device provides mouse functionality includingmoving a cursor; and when the touch mode is activated, manipulate animage displayed on an interactive surface that is separate from themouse-type device in accordance with movement of the mouse-type device.9. The system of claim 8, wherein the mouse-type device comprises amouse button to activate the touch mode.
 10. The system of claim 9,wherein the computer-readable instructions cause the one or moreprocessing devices to: process the movement of the mouse-type device inthe touch mode when the mouse button is depressed, and process themovement of the mouse-type device in the another mode when the mousebutton is not depressed.
 11. The system of claim 9, wherein thecomputer-readable instructions cause the one or more processing devicesto: when the touch mode has been activated by depressing the mousebutton, process the movement of the mouse-type device as a finger dragon the interactive surface that is separate from the mouse-type device.12. The system of claim 8, wherein the image is a digital object that ismanipulated by the movement of the mouse-type device by moving thedigital object, changing a size of the digital object, or rotating thedigital object on the interactive surface that is separate from themouse-type device.
 13. The system of claim 8, wherein thecomputer-readable instructions cause the one or more processing devicesto: when the touch mode is activated, process input received from themouse-type device as a contact point on the interactive surface that isseparate from the mouse-type device.
 14. The system of claim 13, whereinthe computer-readable instructions cause the one or more processingdevices to: when the touch mode is activated, process other inputreceived from a finger of a user as another contact point on theinteractive surface that is separate from the mouse-type device.
 15. Asystem comprising: one or more processing devices; and one or morecomputer-readable memory devices or storage devices storingcomputer-readable instructions which, when executed by the one or moreprocessing devices, cause the one or more processing devices to: detectthat multiple contact points are concurrently engaging a digital object,the multiple contact points concurrently engaging the digital objectincluding a cursor contact point and a finger contact point, wherein thecursor contact point corresponds to input received from an input devicethat controls a cursor and the finger contact point corresponds to afinger; and manipulate the digital object in accordance with themultiple contact points.
 16. The system of claim 15, wherein the inputdevice that controls the cursor is a mouse-type device.
 17. The systemof claim 16, wherein the digital object is manipulated by rotating thedigital object in accordance with movement of the cursor contact pointand movement of the finger contact point.
 18. The system of claim 16,wherein the digital object is manipulated by changing a size of thedigital object in accordance with movement of the cursor contact pointand movement of the finger contact point.
 19. The system of claim 18,further comprising the input device.
 20. The system of claim 19, furthercomprising an interactive surface configured to display the digitalobject.